Computing in High Energy and Nuclear Physics, La Jolla, CA, USA, March 24-28 2003 



1 



Object Database for Constants: The common CLEO Online and Offline 
solution 

N. Adam, S. Lee, H. Schwarthoff, T. Wilksen 
Cornell University, Ithaca, NY 14853, USA 



m ■ 
o ■ 
O ' 

(N : 



(N 



After the successful conclusion of the CLEO III phase, the CLEO experiment at the Cornell electron positron 
storage ring CESR is preparing for its transition to CLEO-c. This new program contains a wide array of Physics 
studies at e + e — collisions at center of mass energies between 3 GeV and 5 GeV that will provide new insights 
into QCD. Because the existing Silicon Vertex Detector needed to be replaced within a short time, a 6 layer 
Vertex Drift Chamber has been installed in Spring 2003. At the same time, the existing Ring Imaging Cherenkov 
Detector, along with a conventional Drift Chamber, E.M. Calorimeter and Muon Chambers, will continue to be 
part of the experiment. 

The CLEO Constants Database system must provide efficient access to detector and analysis constants for a 
wide variety of purposes, such as hardware configuration, Online operation and monitoring, calibration, and the 
complete CLEO analysis framework. Since the original project, implemented through the Objectivity Object 
Database Management System, did not meet those requirements, the system was redesigned from the ground up, 
using the same technology. It is currently being introduced into the production environment without significant 
interruptions. In this presentation, we will outline the specifications of the constants database system, and then 
report on the process that led to the redesign. Performance comparisons and insights on relevant design aspects 
will be shown. 



1. The CLEO experiment 
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The 10.6 GeV e+e~ collider CESR at Cornell Uni- 
versity in Ithaca, New York, has served as an out- 
standing Particle Physics research facility for over 
20 years. The measured peak luminosity is 1.3 x 
10 33 cm -2 s — Its unique detector, constructed and 
operated by the CLEO collaboration, has been able 
to provide a wealth of results from its focus on B me- 
son studies and a variety of other topics such as charm 
and tau physics. Since the year 2000, the CLEO III 
detector ([y) has operated with a 4 layer Si vertex de- 
tector, and a particle identification system based on 
Ring Imaging Cherenkov (RICH) detectors. 

Due to the greatly increased luminosity during the 
CLEO III phase, a new readout and computing system 
was designed to cope with higher demands (0]>[3)- 
Throughout the development process, object oriented 
principles were applied consequently resulting in a co- 
herent and easily extensible set of software compo- 
nents that now cover all aspects of CLEO computing. 
Part of this is the CLEO III database domain, imple- 
mented in an Object Database Management System 
(0). Six individual Online databases for alarms, pro- 
gram code, histograms, run setup data, run statistics 
data, and constants, comprise this system. All are im- 
plemented as C++ database servers, accessible to the 
distributed software components via CORBA commu- 
nication interfaces 

Two Online databases, for run statistics and con- 
stants, are also used in the CLEO Offline analysis 
framework. This framework is used for all CLEO 
physics analysis tasks, such as detector calibration, 
track reconstruction, Monte Carlo data generation, 
and user analysis. It is based on a lightweight on- 
demand data delivery approach, as described else- 



1.1. CLEO-c 

The CLEO III phase has ended in 2003. The stor- 
age ring is in the process of being converted to a charm 
factory (CESR-c,^), adapted for center of mass en- 
ergies between 3 GeV and 5 GeV. It will serve as a 
new research tool for precision charm quark measure- 
ments, searches for new physics, and r lepton stud- 
ies. It is expected to provide an array of Quantum 
Chromo Dynamics (QCD) results that permit com- 
parison to the most recent generation of lattice QCD 
calculations with unprecendented precision. 

Since the original facility was not constructed for 
this energy domain, a set of superconducting wig- 
gler magnets is installed to ensure luminosities around 
2 x lQ 32 cm _2 s _1 . Several of these devices are already 
in operation; the upgrade will be completed in early 
2004. In parallel, the CLEO detector, now called 
CLEO-c, is undergoing modifications to allow opera- 
tion at those lower energies. Unfortunately shortly 
after its deployment, the CLEO III vertex detec- 
tor developed severe particle detection inefficiencies. 
This effect reduced its usability considerably by caus- 
ing limitations in secondary vertex identification. In 
preparation for CLEO-c, it was replaced with a 6 layer 
conventional vertex drift chamber. 



2. The CLEO constants database 

The data structure of a CLEO constants object is 
determined by the user, who writes a definition file in 
c-style syntax, e.g.: 
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Name: RICHChannel 
{ 

UInt32 ChannelAddress 
UIntl6 Crate SelectBy 
UIntl6 Threshold 
UIntl6 Pedestal 
} 

The CLEO build system reads this file and autogener- 
ates a number of source code files for object database 
storage, database server management, CORBA inter- 
facing, client data access, and java classes for Graph- 
ical User Interfaces. This code is compiled and made 
available to all higher level software (Online and Of- 
fline) in libraries and java archives. 

A single constants dataset, called a constants ver- 
sion, consists of one of more instances (lines) of data 
in the form of the described structure, such as the 
four integers in the above example. In CLEO, some 
versions have reached ca. 230000 lines. Every dataset 
receives a unique version number when it is written 
to the database. In addition, a usage tag and run 
validity range can be associated with each constants 
version. 

Users usually do not have write access to the con- 
stants database. Instead, a user can create new con- 
stants objects programmatically, store them as ascii 
file, and then submit them to the central constants 
management system for installation. After applying 
some consistency checks and safeguards, that system 
writes the new data to the database and updates va- 
lidity information. 

After 2 years of operations, the CLEO constants 
database had proven to be a reliable design that 
was extendable for new data types. It became clear 
that separate copies of the full database were needed 
for Online operation, track reconstruction processing, 
public analysis, and remote site Monte Carlo genera- 
tion. For that purpose, the protected Online database 
was designated to be the master database, i.e. the 
only existing copy available for write access. A num- 
ber of copies acted as slave mirrors, readonly versions 
that were updated from the master as needed. This 
scheme allowed minimization of the load on the mas- 
ter, a requirement for smooth and independent Online 
detector operation. 



3. The constants database redesign 

Despite the success in some areas, experience re- 
vealed that the constants database system was de- 
signed in a way that did not match the predominant 
usage patterns. Performance degraded with increased 
database size, causing concern for Online operation. 
Hard disk and CPU contention on the server nodes 
could lead to wait times of minutes for a single re- 
quest. The system could not be expected to scale well 



for the rising number of clients. This situation led to 
the decision to rewrite all constants database access 
code to ensure future scalability and efficiency. 

3.1. Redesign - why? 

A data object that is stored in an object database 
requires a unique identifier (object identifier, OID) 
for data management within the database and for re- 
trieval. Such an object can have almost any shape, 
typically expressed in a c-style structure (or a C++- 
style class). A certain overhead for OID and storage 
is necessary and typically amounts to 20-30 Bytes, 
in addition to the space needed by the object itself. 
Retrieving an object also entails a certain computing 
overhead that is mostly independent of object size. 
From this fact we conclude that it is desirable to keep 
the number of objects retrieved in a single database 
transaction small. 

This notion was not part of the original constants 
database design: A CLEO constants version, with a 
structure as described in the above example, could 
have many lines. The original system architects had 
implemented this as a series of many OIDs, one per 
line. The consequence was considerable overhead in 
storage space and CPU time needed for retrieval: a 
download request always asked for a complete con- 
stants version, i.e. many OIDs. In addition, an in- 
efficient search algorithm that often had to iterate 
through a large number of objects contributed to a 
steady decline in download performance as data vol- 
ume rose over time. 

Another limitation became relevant when new ver- 
sions for one of the existing data types were created in 
greater numbers than originally foreseen. Since every 
object must have a unique OID, the number of objects 
in any part of a database is limited. In this case, there 
could not be more than 2 15 — 1 = 32767 versions. A 
workaround was possible, but only at the expense of 
more complicated data distribution and longer search 
times. 

In the data organization domain, the original sys- 
tem violated a basic rule of database design: Data nor- 
malization: the data contents of a constants version 
was stored not only together with its version number, 
but also with its usage information (usage tag and run 
validity range). In such a setup, one is severely lim- 
ited in how data access can be managed. It is, for 
example, not possible to associate more than one va- 
lidity range - usage tag combination efficiently with an 
object, since a search with a certain tag and run num- 
ber requires examination of many data objects. Since 
CLEO required several such combinations, a cumber- 
some set of separate structures had to be invented to 
provide this functionality. The maintenance of those 
structures was particularly arduous. 

The combination of the described shortcomings 
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Figure 1: Average load time for a single constants version, measured over 10 min intervals. The server side CORBA 
conversion overhead is included. 



eventually provided a strong justification for the re- 
design of the database structure. 

3.2. The new constants database system 

Motivated by a wealth of experiences from the 
CLEO Online and Offline production systems, the 
specifications for the redesign could be determined 
with ease: 

• One OlD per constants version 

• No linear search on retrieval 

• Copious OID space for a large number of ver- 
sions 

• Constants data stored independently of usage 
information 

• Navigational elements that allow flexible usage 
management incorporated into the basic design 

According to these requirements, the new design 
consolidated all lines of one constants version into 
a single database object. This approach reduces 
a database query to the retrieval of one object; 
download pattern and storage structure are exactly 
matched. The new organization now permits very fast 
data retrieval through a set of management structures 
that can be traversed quickly. A lookup with a usage 
tag and run number is organized so that a validity 
list is searched in a tag dictionary, which then holds a 
direct reference to the desired data object. The need 



for linear object searches was completely eliminated 
by using indexed version lists and hash maps for fast 
lookup via dynamic usage tags. Loading an object 
with a specified version number is as easy as open- 
ing a version list and dereferencing it with the version 
number as array index. 

With the number of database objects greatly re- 
duced, the available OID space is now considerably 
larger than needed for the lifetime of the CLEO exper- 
iment. At the same time, the space requirements are 
less than half of the original needs. A remarkable suc- 
cess story of object oriented client-server design was 
also the fact that the client interfaces did not have 
to be modified at all, since the redesign could remain 
completely confined to the database server implemen- 
tation. This allowed a smooth transition of all running 
build and production systems. 

For an illustration of the performance improvement 
that was achieved through the redesign, see Figure ^ 
While the average load time of a single constants ver- 
sion (~ 0.2 s), requested by regular physics reconstruc- 
tion and analysis programs, were already an improve- 
ment from intermediate changes, the new system was 
able to reduce this time by more than a factor of five. 
The distribution is much narrower, which translates 
into a fast, responsive, and reliable behaviour. Since 
a single analysis job can request over 100 different con- 
stants versions, the impact on real world operation is 
dramatic. 
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3.3. Experiences with commercial 
software 

The generic trend in the High Energy and Nuclear 
Physics computing community to purchase commer- 
cial software for their ever growing production systems 
has generated many varied experiences. Physicists 
do not write their own CORBA implementations, but 
they have realized that their database needs may be 
different from what industry requires, and that devel- 
oping solutions on their own is possible and may turn 
out to be the better way. Of concern for the entire 
community is the fact that the widely used commercial 
implementation of an Object Database Management 
System by Objectivity Inc. (H) is being deprecated 
by almost all major collaborations for their large scale 
data storage needs. A variety of reasons is cited: 

• The Physics community is reluctant to adopt 
software without access to the source code, 
since its computing systems are usually highly 
non-standard implementations that require cus- 
tomization. 

• An object database does not match traditional 
data access and usage patterns. 

• Large scale data storage with an object database 
requires a highly sophisticated and finetuned 
setup and time consuming expert operation. 

• Financial means in public research are limited. 

Although those issues are of little direct consequence 
for the project described here, because of the much 
smaller database sizes, a decline in overall acceptance 
will have an impact on product pricing, availability, 
and support. 

Similarly, the situation continues to be challenging 
with regard to platform support. Third party software 
packages, such as CORBA implementations, make in- 
roads into the Linux world at a slow pace. But the 
Linux on Intel platform is in the process of displac- 
ing all other hardware/operating system combinations 
in the field. This situation has placed limitations 



on what can be done with commercial software, and 
may prove a confining factor in future upgrades of the 
CLEO computing systems. 
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